ML Report Descriptions:

The two main goals with this document are to provide a wide range of output to investigate high level performance and insights, and to deliver a high quality report design layout to increase user experience. The metrics provided are intended to be semi-comprehensive. One can always dig deeper into results to gain further insights. In light of that, the results are intended to provide information that one can come to a reasonable conclusion about their model or to find the areas where they need to dig a little deeper.

Evaluation Metrics

This section contains statistics and variable importance measures to help the user understand model performance at a high level. Train Data results are included and can be used to compare against the Test Data results to identify over / under fitting of models.

Evaluation Plots

This section contains visualizations that span the range of predicted values and the associated accuracies across that range. The predicted values range is broken up into every 5th percentile to provide a wide range for evaluation.

Model Interpretation sections

This section contains visualizations intended to open up the black box of your algorithm. When one inspects coefficients from a regression model, the insights they gain are two-fold: - get an understanding about statistics significance - gain an understanding of the variable’s effect on the target variable However, not all relationships are linear and sometimes the user doesn’t specifiy an appropriate model structure to fully capture the nature of the relationship, which can lead to incorrect conclusions about both statistical signifance and the nature of the relationship. These visualizations provide a way to understand what the exact nature of those relationships are (in a visual manner) and if the user chooses, they can attempt to fit the relationship more precisely with an appropriate statistical model in order to gain a better understanding of statistical significance.

Evaluation Metrics

Expand

MultiClass Metrics Tables

MultiClass Metrics

Binary Metrics 1 vs All Tables

Model Metrics Tables

TestData

Performance Metrics

## [1] "TestData_I"
## [1] "TestData_J"
## [1] "TestData_O"
## [1] "TestData_U"
## [1] "TestData_Y"

TrainData + ValidationData

Performance Metrics

## [1] "TrainData_I"
## [1] "TrainData_J"
## [1] "TrainData_O"
## [1] "TrainData_U"
## [1] "TrainData_Y"

Variable Importance Table

Variable Importance

Interaction Importance Table

Interaction Importance

## [1] "Interaction importance is only available with CatBoost"

Evaluation Plots

Expand

Variable Importance Plots

Expand

Calibration Plots

Expand

TestData

Calibration Plot

TrainData + ValidationData

Calibration Plot

ROC Plots

Expand

TestData

ROC Plots

TrainData + ValidationData

Expand

Lift & Gains Plots

Expand

TestData

Lift & Gains Plots

TrainData + ValidationData

Expand

Model Interpretation

Expand

Partial Dependence Plots: Numeric-Features

Expand

Partial Dependence Line Plots

Expand

TestData

Partital Dependence Line Plots

TrainData + ValidationData

Partital Dependence Line Plots

Partial Partial Dependence Plots: Categorical-Features

Expand

TestData

Partital Dependence Bar Plots

TrainData + ValidationData

Partital Dependence Bar Plots

Model MetaData

Expand

Parameters and Settings

Model Parameters

## [1] "HasTime: FALSE"
## [1] "EvalMetric: "
## [1] "LossFunction: MultiClassOneVsAll"
## [1] "OutputSelection: Importances"     "OutputSelection: EvalPlots"      
## [3] "OutputSelection: EvalMetrics"     "OutputSelection: Score_TrainData"
## [1] "TargetColumnName: Adrian"
##  [1] "FeatureColNames: Independent_Variable1" 
##  [2] "FeatureColNames: Independent_Variable2" 
##  [3] "FeatureColNames: Independent_Variable3" 
##  [4] "FeatureColNames: Independent_Variable4" 
##  [5] "FeatureColNames: Independent_Variable5" 
##  [6] "FeatureColNames: Independent_Variable6" 
##  [7] "FeatureColNames: Independent_Variable7" 
##  [8] "FeatureColNames: Independent_Variable8" 
##  [9] "FeatureColNames: Independent_Variable9" 
## [10] "FeatureColNames: Independent_Variable10"
## [11] "FeatureColNames: Factor_2"              
## [1] "PrimaryDateColumn: "
## [1] "WeightsColumnName: "
## [1] "IDcols: IDcol_1" "IDcols: IDcol_2"
## [1] "EncodeMethod: catboost"
## [1] "TrainOnFull: FALSE"
## [1] "task_type: GPU"
## [1] "NumGPUs: 0"
## [1] "DebugMode: FALSE"
## [1] "ReturnModelObjects: TRUE"
## [1] "SaveModelObjects: FALSE"
## [1] "ModelID: Test_Model_1"
## [1] "model_path: C:/Users/Bizon/Documents/GitHub/AutoQuant"
## [1] "metadata_path: C:/Users/Bizon/Documents/GitHub/AutoQuant"
## [1] "ClassWeights: "
## [1] "NumOfParDepPlots: 11"
## [1] "eval_metric: MultiClassOneVsAll"
## [1] "loss_function: MultiClassOneVsAll"
## [1] "grid_eval_metric: Accuracy"
## [1] "BaselineComparison: default"
## [1] "MetricPeriods: 10"
## [1] "PassInGrid: "
## [1] "GridTune: FALSE"
## [1] "MaxModelsInGrid: 30"
## [1] "MaxRunsWithoutNewWinner: 20"
## [1] "MaxRunMinutes: 1440"
## [1] "Trees: 50"
## [1] "Depth: 6"
## [1] "LearningRate: "
## [1] "L2_Leaf_Reg: "
## [1] "RandomStrength: 1"
## [1] "BorderCount: 128"
## [1] "RSM: "
## [1] "BootStrapType: Bayesian"
## [1] "GrowPolicy: SymmetricTree"
## [1] "langevin: FALSE"
## [1] "diffusion_temperature: 10000"
## [1] "model_size_reg: 0.5"
## [1] "feature_border_type: GreedyLogSum"
## [1] "sampling_unit: Object"
## [1] "subsample: "
## [1] "score_function: Cosine"
## [1] "min_data_in_leaf: 1"
## [1] "Algo: CatBoost"
## [1] "TargetType: MultiClass"
## [1] "PredictionColumnName: Predict"
## [1] "TargetLevels: c(\"I\", \"J\", \"O\", \"U\", \"Y\")"
## [2] "TargetLevels: 1:5"

Grid Tuning Metrics

Grid Tuning Metrics

## [1] "GridTuning was not conducted"